Information processing apparatus, information processing system, information processing method, and program

ABSTRACT

An information processing apparatus is connected to a first information acquisition apparatus that acquires first information relating to at least one of a position or a pose of a hand of a target person and a second information acquisition apparatus that acquires second information relating to at least one of the position or the pose of the hand of the target person and different from the first information acquired by the first information acquisition apparatus. The information processing apparatus accepts the first information and the second information from the first and second information acquisition apparatuses, respectively, retains the accepted first and second information in an associated relationship with information of timings at which the first and second information acquisition apparatuses acquire the first and second information, respectively, and extracts pieces of the first and second information acquired at a common timing from the retained first and second information as pair information.

TECHNICAL FIELD

The present invention relates to an information processing apparatus, aninformation processing system, an information processing method, and aprogram.

BACKGROUND ART

In recent years, an information processing apparatus such as a consumergame machine that recognizes a position or a pose of a hand of a playerand provides information of the recognized position or pose of the handin processing of a game or the like has been and is being developed.

In such an information processing apparatus as just described, a deviceto be grasped by a hand of a player (hereinafter referred to as acontroller device) measures, for example, a distance from the controllerdevice itself to fingers of the player or the like. Then, the positionor the pose of the hand of the player is estimated using the informationof the measured distances between the controller device and the fingers,information of a posture or the like of the controller device itself andso forth.

In a different example, the information processing apparatus captures animage of a hand of a player by a camera or the like and estimates theposition or the pose of the hand of the player on the basis of thecaptured image of the hand.

While various methods are available for an estimation process, forexample, a method using a result of mechanical learning of artificialintelligence or the like is available as an example. In the case where aresult of mechanical learning is used, for example, information of aposition or a pose of a hand of a user that is a correct answer ismeasured and acquired by a device (hereinafter referred to as ameasurement device for the convenience of description) other than adevice such as the controller device or the camera (hereinafter referredto as a supposing device for the convenience of description). Then, aneural network is learned using the correct answer information as ateacher signal and using the information acquired by the controllerdevice as an input signal, and then the neural network is used.

SUMMARY Technical Problems

However, it is not necessarily easy to acquire correct answerinformation to be used as the teacher signal and find out informationacquired by a controller device corresponding to the correct answerinformation.

For example, in the case where information of a position or a pose of ahand of a target person for measurement is acquired as correct answerinformation using the measurement device described above, it isnecessary to find out, corresponding to a point of time at which theinformation is acquired, information acquired by the supposing device(an image of a camera, information of a distance between the controllerdevice and the fingers or the like). However, according to aconventional method, the target person is urged to move its hand to adesignated position and take a designated pose, information is acquiredby the measurement device, and then the information acquired by thesupposing device is recorded in an associated relationship at the pointof time. Then, pieces of information that relate to one by one positionand pose, are associated with each other and are to be acquired by themeasurement device (teacher signal) and information acquired by thesupposing device (input signal) are set by a manual work, and then amechanical learning process is performed.

Therefore, the burden when data for mechanical learning (informationthat associates the teacher signal and the corresponding input signalwith each other) is prepared is great.

The present invention has been made in view of such a situation asdescribed above, and it is one of objects of the present invention toprovide an information processing apparatus, an information processingsystem, an information processing method, and a program by which datafor mechanical learning for estimating information of a position or apose of a hand of a target person or the like can be generatedefficiently.

Solution to Problems

One aspect of the present invention that solves the problems of theconventional examples described above is an information processingapparatus connected to a first information acquisition apparatus thatacquires first information relating to at least one of a position or apose of a hand of a target person and a second information acquisitionapparatus that acquires second information relating to at least one ofthe position or the pose of the hand of the target person and differentfrom the first information acquired by the first information acquisitionapparatus, and including acceptance means for accepting the firstinformation and the second information from the first and secondinformation acquisition apparatuses, respectively, retention means forretaining the accepted first and second information in an associatedrelationship with information of timings at which the first and secondinformation acquisition apparatuses acquire the first and secondinformation, respectively, and extraction means for extracting pieces ofthe first and second information acquired at a common timing from theretained first and second information as pair information, and in whichthe extracted pair information is provided for a learning process ofmechanical learning using the first and second information included inthe pair information as a teacher signal and an input signal,respectively.

Advantageous Effects of Invention

With the present invention, data for mechanical learning for estimatinginformation of a position or a pose of a hand of a target person can beproduced efficiently.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram depicting an example of a configuration of aninformation processing system according to an embodiment of the presentinvention.

FIG. 2 is an overview explanatory view depicting an example of ameasurement device used by the information processing system accordingto the embodiment of the present invention.

FIG. 3 is an outline explanatory view depicting an example of anestimation device used by the information processing system according tothe embodiment of the present invention.

FIG. 4 is a functional block diagram depicting an example of aninformation processing apparatus according to the embodiment of thepresent invention.

FIG. 5 is an explanatory view depicting an example of informationretained by the information processing apparatus according to theembodiment of the present invention.

FIG. 6 is a flow chart depicting an example of operation of informationprocessing according to the embodiment of the present invention.

DESCRIPTION OF EMBODIMENT

In the following, an embodiment of the present invention is describedwith reference to the drawings. As exemplified in FIG. 1, an informationprocessing system 1 according to the embodiment of the present inventionis configured by including a measurement device 10 as a firstinformation acquisition apparatus, an estimation device 20 as a secondinformation acquisition device and an information processing apparatus30.

Here, as depicted in FIG. 1, the information processing apparatus 30 isconfigured by including a control unit 31, a storage unit 32, anoperation unit 33, a display controlling unit 34, and an inputting andoutputting unit 35.

Further, as exemplified in FIG. 2, the measurement device 10 of oneexample of the present embodiment includes a large number of cameras Carranged around a hand of a target person who is a target ofmeasurement, and can capture an image of the hand by a plurality ofcameras C1, C2, . . . from among the cameras C at whichever position andwhatever pose the hand of the target person takes. Further, themeasurement device 10 includes an information processing device such asa PC (personal computer), and recognizes, from an image captured atevery predetermined timing (for example, at every periodical timing suchas every 1/60 second), the position of each of a plurality of markers,which are pasted to main positions of a hand (for example, at jointpositions of the fingers or the like) of the target person, in athree-dimensional space. Then, the measurement device 10 outputsinformation of the recognized position (three-dimensional coordinateinformation) of each marker together with timestamp informationrepresentative of the point of time of imaging. Since such a device asjust described can be implemented using a system known as a motioncapture system such as OptiTrack (registered trademark) by SPICE INC.,detailed description of the device is omitted here. Naturally, themeasurement device may use any method that uses electromyography,ultrasonic waves or the like only if the measurement device can acquireat least one of a position or a pose of a hand of a target personwithout fail.

Further, the estimation device 20 according to one example of thepresent embodiment is a controller device 200 that is grasped by a handof a target person, for example, as exemplified in FIG. 3. Thiscontroller device 200 has, in the example of FIG. 3, an elongatedcolumn-like shape and has, at positions thereof with which the fingersof the target person when the controller device 200 is grasped by thetarget person are to contact, sensors 201 for measuring the position toa fingertip such as capacitance sensors or infrared distance sensors.

It is to be noted that, in the examples of FIGS. 2 and 3, the hand ofthe target person is indicated by broken lines for the convenience ofdescription.

In the example of the present embodiment, the controller device 200repeatedly acquires information of the distance to a fingertip of thetarget person measured by each sensor 201 at every predetermined timing(for example, at every periodical timing after every 100 milliseconds orthe like). Then, every time such information is acquired, the controllerdevice 200 sends out the information to the information processingapparatus 30.

The control unit 31 of the information processing apparatus 30 is aprogram-controlled device such as a CPU (central processing unit) andoperates in accordance with a program stored in the storage unit 32. Thecontrol unit 31 in the present embodiment accepts three-dimensionalcoordinate information representative of the positions of markersarranged at different portions of a hand of the target person andtimestamp information representative of a point of time of capturing animage of the markers as first information from the measurement device 10that is the first information acquisition apparatus. Further, thecontrol unit 31 accepts information acquired by the estimation device 20from the estimation device 20 as the second information acquisitionapparatus. For example, if the estimation device 20 is the controllerdevice 200, then the control unit 31 accepts information of the distanceto the fingertips of the target person as second information from thecontroller device 200 at every predetermined timing (for example, atevery periodical timing after every 100 milliseconds or the like).

The control unit 31 retains the accepted first and second information inan associated relationship with information of timings at which thefirst and second information is acquired by the measurement device 10and the estimation device 20 that are the first and second informationacquisition apparatuses, respectively. Further, the control unit 31extracts pieces of the first and second information acquired at commontimings from among the retained first and second information as pairinformation. Then, the extracted pair information is used in a learningprocess of mechanical learning in which the first information and thesecond information included in the pair information are used as ateacher signal and an input signal, respectively.

The storage unit 32 is a memory device, a disk device or a like deviceand retains a program to be executed by the control unit 31. Thisprogram is stored in and provided as a computer-readable non-transitoryrecording medium and may be stored in the storage unit 32. Further, thestorage unit 32 operates also as a work memory of the control unit 31.

The operation unit 33 is connected to a device such as a keyboard or amouse, and accepts an input of an instruction operation of a user fromthe device and outputs information representative of the content of theinstruction operation to the control unit 31. The display controllingunit 34 is connected to a display or a like device and controls thedisplay or the like device to display and output information inaccordance with an instruction inputted from the control unit 31.

The inputting and outputting unit 35 is connected to the measurementdevice 10 and the estimation device 20 by wired communication by USB(universal serial bus) or the like or wireless communication byBluetooth (registered trademark) communication, and accepts informationoutputted from the devices and outputs the information to the controlunit 31. Further, the inputting and outputting unit 35 outputsinformation to a device connected thereto in accordance with aninstruction inputted from the control unit 31.

Now, operation of the control unit 31 is described. The control unit 31of the present embodiment is configured by functionally including anacceptance unit 41, a pre-processing unit 42, a holding processing unit43 and an extraction processing unit 44 as exemplified in FIG. 4. It isto be noted that, in the example described below, it is assumed that theestimation device 20 connected to the information processing apparatus30 is the controller device 200.

The acceptance unit 41 accepts first information from the measurementdevice 10 that is the first information acquisition apparatus throughthe inputting and outputting unit 35. Here, the first informationincludes three-dimensional coordinate information (P1, P2, . . . )representative of positions, which are recognized on the basis of animage captured at every predetermined timing, of the markers arranged atportions of the hand of the target person and timestamp information (T)representative of a point of time at which an image of the markers iscaptured.

Further, the acceptance unit 41 accepts information (FP1, FP2, . . . )of the distance between the controller device 200 and the fingertip ofeach finger of the target person at every predetermined timing (at aperiodical timing such as every 100 milliseconds) as second informationfrom the controller device 200 as the second information acquisitionapparatus through the inputting and outputting unit 35.

The pre-processing unit 42 executes a pre-process for retaining firstand second information accepted by the acceptance unit 41 by the holdingprocessing unit 43. In a certain example of the present embodiment, thepre-processing unit 42 acquires, every time the pre-processing unit 42accepts the second information from the controller device 200,information of a timing (time information) at which the secondinformation is accepted. It is sufficient if the time information isacquired from a time counting circuit (RTC (real time clock) or a likecircuit) not depicted or acquired from a time information server througha network or the like.

The pre-processing unit 42 subtracts reference time t0, which is definedby time information t acquired when second information is accepted forthe first time after processing is started, from the time information tacquired at the time of the second information accepted for the i-thtime to obtain timing information ti (i=1, 2, . . . ; accordingly, forthe second information for the first time, t1=0). Then, thepre-processing unit 42 places the timing information ti into acorresponding piece of the second information and outputs the piece ofthe second information to the holding processing unit 43. Further, inthe present example, the pre-processing unit 42 outputs the firstinformation as it is to the holding processing unit 43.

The holding processing unit 43 accepts the first and second information(second information here includes the timing information) outputted fromthe pre-processing unit 42 and cumulatively stores the information intothe storage unit 32.

By this operation of the holding processing unit 43, in regard to thefirst information, the storage unit 32 stores, at each point of time atwhich an image of the markers represented by the timestamp information(T1, T2, . . . ) is captured, three-dimensional coordinate information(P1, P2, . . . ) representative of the positions of the markers obtainedon the basis of the markers whose images are captured at the point oftime in an associated relationship with the point of time as exemplifiedin FIG. 5.

Meanwhile, in regard to the second information, at each point of timerepresented by the timing information (t1, t2, . . . ) obtained by thepre-processing unit 42, information, which is accepted at the point oftime, of the distance (FP1, FP2, . . . ) between the controller device200 and the fingertip of each finger of the target person iscumulatively stored into the storage unit 32 in an associatedrelationship with the point of time.

The extraction processing unit 44 extracts pieces of the first andsecond information acquired at a common timing from among the first andsecond information retained in the storage unit 32 as pair information.In particular, the extraction processing unit 44 selects, for example,pieces of information with an interval between timings at which therecorded information is obtained between the first and secondinformation. For example, if the first information is acquired afterevery ΔT= 1/60 second (approximately 17 milliseconds) and the secondinformation is acquired after every Δt=100 milliseconds, then sinceΔT<Δt is satisfied, the extraction processing unit 44 selects the secondinformation.

The extraction processing unit 44 reads out information retained in anassociated relationship with one piece of timing information from amongthe selected second information from the storage unit 32. In particular,the extraction processing unit 44 selects one piece of timinginformation ti (i=1, 2, . . . ) and reads out information (FP1, FP2, . .. ) of the distance to the finger tip of each finger of the targetperson associated with the selected timing information t1.

The extraction processing unit 44 searches the storage unit 32 for thefirst information (information with regard to which the interval betweentimings at which the recorded information is obtained is comparativelyshort) associated with the information T representative of a timingnearest to time ti represented by the selected timing information.

In the example here, the information of a timing included in the firstinformation is a timestamp, and in regard to the second information,information of a timing corresponding to an elapsed period of time aftermeasurement is started is associated. Therefore, the extractionprocessing unit 44 of this example of the present embodiment determinesthat the time point T0 of a timestamp representing time earliest in timeis equal to the reference time point t0 acquired by the pre-processingunit 42 when the second information is accepted. Thus, the extractionprocessing unit 44 finds out, from among the values Tj−T0 (j=1, 2, . . .) when T0 is subtracted from the time stamps Tj (j=1, 2, . . . )included in the first information, the timestamp Tj nearest to the timeti represented by the selected timing information.

The extraction processing unit 44 reads out the three-dimensionalinformation (P1, P2, . . . ) representative of the positions of themakers arranged at the portions of the hand of the target personassociated with the timestamp Tj found out as described above from thestorage unit 32. Further, the extraction processing unit 44 reads outthe information of the distance (FP1, FP2, . . . ) between the fingertipof each finger of the target person and the controller device 200 storedin the storage unit 32 in which the selected timing information isstored in an associated relationship with the time ti represented by theselected timing information.

The extraction processing unit 44 uses the three-dimensional coordinateinformation (P1, P2, . . . ) representative of the positions of themarkers read out here as a teacher signal representative of a correctanswer and uses the information of the distance (FP1, FP2, . . . )between the fingertip of each finger of the target person and thecontroller device 200 as an input signal. Then, the extractionprocessing unit 44 cumulatively stores the teacher signal and the inputsignal in a set as learning information into the storage unit 32.

The extraction processing unit 44 repeats the processes described abovefor each piece of the timing information ti (i=1, 2, . . . ) of thesecond information to cumulatively store learning information into thestorage unit 32.

The control unit 31 in the present embodiment generates learninginformation in such a manner as described above and uses the learninginformation cumulatively stored in the storage unit 32 in this mannerfor a mechanical learning process of a neural network and so forth.Since a suitable one of various widely known methods can be adopted asthe method for performing a mechanical learning process on the basis ofa set of a teacher signal and an input signal, detailed description ofthe method is omitted here.

[Operation] The information processing system 1 according to the oneexample of the present embodiment has the foregoing configuration, andthe information processing system 1 according to this example operatesin the following manner. A measurement target person makes such actionas to paste markers at main positions of its hand (for example, jointpositions of the fingers or the like) and arbitrarily moves the hand orthe fingers as designated in advance in a state in which the controllerdevice 200 is grasped to move the position of the hand or change thepose.

The measurement device 10 captures an image of the hand of the targetperson from a plurality of places at every predetermined timing (forexample, at every periodical timing after 1/60 second or the like) andrecognizes the positions in the three-dimensional space of the pluralityof markers pasted to the hand of the target person. Then, themeasurement device 10 outputs information of the positions(three-dimensional coordinate information) of the markers together withtimestamp information representative of the point of time of theimaging. It is to be noted that the timing of the outputting may bedetermined such that, after the target person ends all motions, theinformation is outputted collectively.

Further, the controller device 200 grasped by the hand of the targetperson repeatedly acquires the information of the distance to eachfingertip of the target person measured by the sensors 201 arranged onthe surface of the controller device 200 at every predetermined timing(for example, at every periodical timing after 100 milliseconds or thelike) and sends out the information to the information processingapparatus 30 every time the controller device 200 acquires theinformation.

The information processing apparatus 30 accepts, every time thecontroller device 200 accepts the information (FP1, FP2, . . . ) of thedistance between the controller device 200 and the fingertip of eachfinger of the target person, the information as second information fromthe controller device 200.

Then, every time the second information is accepted from the controllerdevice 200, the information processing apparatus 30 acquires informationof the timing (time information) at which the second information isaccepted. The information processing apparatus 30 subtracts referencetime t0, which is defined by time information t acquired when the secondinformation is accepted for the first time, from the time information tacquired at the time of the second information accepted for the i-thtime to obtain timing information ti (i=1, 2, . . . ; accordingly, forthe second information for the first time, t1=0). Then, the informationprocessing apparatus 30 places the timing information ti into acorresponding piece of the second information. The informationprocessing apparatus 30 cumulatively stores the second information inwhich the timing information is included into the storage unit 32.

On the other hand, the information processing apparatus 30 accepts theinformation of the positions (three-dimensional coordinate information)of the markers acquired at every predetermined timing by the measurementdevice 10 during the measurement together with the timestampinformation, for example, when the measurement ends. Then, theinformation processing apparatus 30 cumulatively stores thecorresponding timestamp information and the information (P1, P2, . . . )of the positions of the markers in an associated relationship with eachother into the storage unit 32.

In this manner, the storage unit 32 is placed in a state in which, asexemplified in FIG. 5, in regard to the first information, for eachpoint of time at which an image of the markers represented by thetimestamp information (T1, T2, . . . ) is captured, three-dimensionalcoordinate information (P1, P2, . . . ) obtained on the basis of theimage of the markers captured at the point of time and representative ofthe positions of the markers is stored in an associated relationshiptherewith. In regard to the second information, for each point of timerepresented by the timing information (t1, t2, . . . ), information(FP1, FP2, . . . ), accepted at the point of time, of the distancebetween the controller device 200 and the fingertip of each finger ofthe target person is cumulatively stored in the storage unit 32 in anassociated relationship therewith.

The information processing apparatus 30 accepts an instruction togenerate learning information for mechanical learning, for example, froma user, and starts processing for generating the learning information.The information processing apparatus 30 performs the following processeswith reference to any one of the first and second information. Here, itis sufficient if information in regard to which the distance betweentimings at which recorded information is obtained is comparatively long(information in regard to which the number of obtained data is smaller)is selected. The following description proceeds assuming that the secondinformation is selected.

As exemplified in FIG. 6, the information processing apparatus 30acquires a time point T0 of a timestamp representative of time earliestin time from among the timestamps stored in the storage unit 32 inregard to the first information (S1).

The information processing apparatus 30 successively selects the piecesof the timing information (t1, t2, . . . ) included in the selectedsecond information to repeatedly execute the following processes (S2).In particular, the information processing apparatus 30 searches thestorage unit 32 for the first information associated with theinformation T representative of a timing nearest to the selected timinginformation ti (information in regard to which the distance betweentimings at which the recorded information is obtained is comparativelyshort).

In particular, the information processing apparatus 30 subtracts T0 fromthe timestamps Tj (j=1, 2, . . . ) stored in the storage unit 32 toobtain values Tj−T0 (j=1, 2, . . . ) and finds out the timestamp Tj ofthe value Tj−T0 nearest to the timing information ti (S3).

Then, the information processing apparatus 30 reads out thethree-dimensional coordinate information (P1, P2, . . . ) representativeof the positions of the markers associated with the timestamp Tj foundout in the process S3 (S4).

Further, the information processing apparatus 30 reads out theinformation (FP1, FP2, . . . ) of the distance between the fingertip ofeach finger of the target person and the controller device 200associated with the timing information ti selected by the process S2(S5).

The information processing apparatus 30 uses the three-dimensionalcoordinate information (P1, P2 . . . ) representative of the positionsof the markers read out in the process S4 as a teacher signalrepresentative of a correct answer and uses the information of thedistance (FP1, FP2, . . . ) between the fingertip of each finger of thetarget person and the controller device 200 read out in the process S5as an input signal. Then, the information processing apparatus 30cumulatively stores the signals in a set as learning information intothe storage unit 32 (S6).

The information processing apparatus 30 repeatedly executes theprocesses described above beginning with the process S2 until no pieceof timing information (t1, t2, . . . ) that has not been selected as yetremains anymore. Consequently, the storage unit 32 is placed into astate in which a set of a teacher signal and an input signal (pairinformation) for each piece of timing information (t1, t2, . . . )representative of a timing at which the second information is obtainedis retained.

In this manner, according to the present embodiment, the necessity forcombination of a teacher signal and an input signal by manual work iseliminated, and data for mechanical learning for estimating informationof a position, a pose or the like of a hand of the target person can begenerated efficiently.

[Example in which the content of information is referred to] In thedescription above, in order to find out first and second informationacquired at a common timing, information of a timing representative of apoint of time at which the second information is acquired is associatedwith the second information. Then, an example is described in which,presupposing that a start point of time of measurement is common to themeasurement device 10 and the estimation device 20, first informationassociated with a timestamp representative of a point of time nearest toa point of time represented by the associated timing information issearched for.

However, in the case where there is a difference between a measurementstart point of time of the measurement device 10 and a point of time atwhich information is outputted for the first time from the estimationdevice 20, an error occurs with the information for learning. Therefore,in one example of the present embodiment, a user who is to generatelearning information may visually confirm first information and secondinformation and designate a point of time at which both of theinformation designate a same position or pose of the hand to theinformation processing apparatus 30.

For example, in the case where information at a timestamp Tk of thefirst information and a piece of information in the timing informationtm from among the second information represent a same position or poseof the hand, the pieces of information are designated. Consequently, theinformation processing apparatus 30 acquires the difference betweenΔTk=Tk−T0 (here, T0 is information of a timestamp representative of timeearliest in time from among the timestamps stored in the storage unit 32in regard to the first information described hereinabove) and tm, thatis, Δ=ΔTk−tm, as correction information. Then, in the process S3described above, the information processing apparatus 30 calculatesvalues Tj−T0 (j=, 1, 2, . . . ) by subtracting T0 from the timestamps Tj(j=1, 2, . . . ) stored in the storage unit 32 and finds out a timestampTj having a value Tj−T0 nearest to a value obtained by adding thecorrection information Δ to the timing information ti, that is, to ti+Δ.

Further, although it is described here that the user visually searchesfor a point of time at which the first and second information representsa same position or pose of the hand, this search may be mechanicallyperformed (without any artificial processing) by the informationprocessing apparatus 30.

In this case, for example, the target person is caused to initiallyperform a motion of holding a hand (establishing a state in which thetarget person touches at the fingertips thereof with the controllerdevice 200) and a motion of opening the hand (establishing a state inwhich the fingertips are spaced from the controller device 200 as far aspossible). Then, the information processing apparatus 30 obtainsinformation Tmin of a timestamp associated with a piece of the firstinformation, which is located before a time point Tf at which, when therecorded first information is checked in the ascending order of thetimestamp, information representing that the target person is in a statein which the hand is open (for example, information with which thepoints of three-dimensional coordinates of the markers pasted to theindividual fingers are located substantially linearly) is recorded forthe first time and which represents that the target person is in a statein which the hand is held (for example, a piece of the first informationwhen the volume of a minimum projecting hull including a point of thethree-dimensional coordinates of each marker is smallest).

Further, the information processing apparatus 30 checks the recordedsecond information in a recording order (in the ascending order of timerepresented by the information of the associated timings) to obtaininformation tmin of the timing associated with the informationrepresenting that the target person is in a state in which the hand isheld and found out for the first time in the check (for example, thesecond information in which the distance to each fingertip is in theminimum).

Then, the information processing apparatus 30 acquires the differencebetween ΔTk=Tmin−T0 (here, information of the timestamp representativeof time earliest in time from among the timestamps stored in the storageunit 32 in regard to the first information described hereinabove) andtmin, that is, Δ=ΔTk−tmin, as correction information. Then, in theprocess S3 described above, the information processing apparatus 30subtracts T0 from the timestamps Tj (j=1, 2, . . . ) stored in thestorage unit 32 to obtain values Tj−T0 (j=1, 2, . . . and finds out atimestamp Tj having a value Tj−T0 nearest to the value obtained byadding the correction information Δ to the timing information ti, thatis, ti+Δ.

Consequently, determining a timing at which the position or the pose ofthe hand of the target person represented by the retained firstinformation is determined to be substantially same as the position orthe pose of the hand of the target person represented by the retainedsecond information as a common timing, pieces of the first and secondinformation acquired at the common timing can be extracted as pairinformation.

[Example in which a marker is pasted also to an estimation device]Further, it is described in the foregoing description that the markersdetected by the measurement device 10 that is the first informationacquisition apparatus are pasted to portions of a hand of a targetperson, in an example of the present embodiment, the markers may beapplied also to the controller device 200 that is the second informationacquisition apparatus.

In this case, the measurement device 10 acquires information of theposition, posture and so forth of the controller device 200 as theestimation device 20 together with information relating to a position ora pose of the hand of the target person. Then, the informationprocessing apparatus 30 records first information including theinformation of the position and the posture of the controller device 200together with the information relating to the position and the pose ofthe hand of the target person. Further, the information processingapparatus 30 extracts a teacher signal from the record of the firstinformation and extracts the second information (input signal) acquiredby the controller device 200 at a timing common to the timing at whichthe extracted teacher signal is obtained, and then outputs the teachersignal and the input signal as pair information.

In this example of the present embodiment, a mechanical learning processfor estimating not only the position or the pose of a hand of a playerwho grasps the controller device 200 but also information of theposition or the posture of the grasped controller device 200 becomespossible.

[Example in which the estimation device is a camera] Further, in thepresent embodiment, the estimation device 20 may not necessarily be thecontroller device 200 that is grasped by a hand of a target person. Forexample, in a certain example of the present embodiment, the estimationdevice 20 may be a depth camera including a pair of cameras arranged ata predetermined one position for a player (target person) such that thecameras are directed toward the target person.

In this case, depending upon the situation of the posture or the pose ofa hand of the target person, the depth camera that is the estimationdevice 20 cannot necessarily view all fingers of the target person (forexample, in such a case where the back of a hand of a player enters aplace from the depth camera to the fingers). However, the measurementdevice 10 can acquire position information of all markers pasted to thejoints and so forth of all the fingers of the target person.

In this example, the second information includes image data of twoimages captured by the pair of cameras included in the depth cameras.

Further, in this example, every time the acceptance unit 41 acceptssecond information, the pre-processing unit 42 acquires information ofthe timing (time information) at which the second information isaccepted. Further, the pre-processing unit 42 subtracts reference timet0, which is defined by time information t acquired when secondinformation is accepted for the first time after processing is started,from the time information t acquired at the time of the secondinformation accepted for the i-th time to obtain timing information ti(i=1, 2, . . . ; accordingly, for the second information for the firsttime, t1=0). Then, the pre-processing unit 42 places the timinginformation ti into a corresponding piece of the second information andoutputs the piece of the second information to the holding processingunit 43.

Furthermore, in this example of the present embodiment, thepre-processing unit 42 may perform a process for removing an image ofmarkers for the recognition by the measurement device 10, which isincluded in image data that is second information. In particular, sincethe markers are colored, for example, in a color determined in advance,the pre-processing unit 42 performs a process for extracting the coloredportions of the markers included in the image data that is the secondinformation and filling the portions with a color surrounding theportions. Since a process widely known as an image process of photoretouching or the like can be used for the process mentioned, detaileddescription is omitted herein.

If the pre-process is performed in this manner, the input signal forlearning information is placed into a state in which an image thatincludes no marker is provided. Usually, in a scene in which a neuralnetwork or the like learned actually using learning information isutilized, image data captured by the depth camera does not include anymarker. Therefore, since the input signal that is learning informationis made proximate to the actually utilized image data, it is expectedthat more precise estimation is performed.

REFERENCE SIGNS LIST

1 Information processing system, 10 Measurement device, 20 Estimationdevice, 30 Information processing apparatus, 31 Control unit, 32 Storageunit, 33 Operation unit, 34 Display controlling unit, 35 Inputting andoutputting unit, 41 Acceptance unit, 42 Pre-processing unit, 43 Holdingprocessing unit, 44 Extraction processing unit, 200 Controller device,201 Sensor.

The invention claimed is:
 1. An information processing apparatusconnected to a first information acquisition apparatus that acquiresfirst information relating to at least one of a position or a pose of ahand of a target person and a second information acquisition apparatusthat acquires second information relating to at least one of theposition or the pose of the hand of the target person and different fromthe first information acquired by the first information acquisitionapparatus, the information processing apparatus comprising: anacceptance unit for accepting the first information and the secondinformation from the first and second information acquisitionapparatuses, respectively; a retention unit for retaining the acceptedfirst and second information in an associated relationship withinformation of timings at which the first and second informationacquisition apparatuses acquire the first and second information,respectively; and an extraction unit for extracting pieces of the firstand second information acquired at a common timing from the retainedfirst and second information as pair information, wherein the extractedpair information is provided for a learning process of mechanicallearning using the first and second information included in the pairinformation as a teacher signal and an input signal, respectively, andwhen it is known that, with regard to a particular position or pose ofthe hand of the target person, first information pieces relating to theparticular position or pose and the second information pieces relatingto the particular position or pose are not acquired at a common timing,a correction is applied to nevertheless extract the pieces as pairinformation.
 2. The information processing apparatus according to claim1, wherein the extraction unit extracts, on a basis of a content of theretained first and second information, pieces of the first and secondinformation acquired at the common timing as the pair information. 3.The information processing apparatus according to claim 2, wherein theextraction unit determines a timing at which the position or the pose ofthe hand of the target person represented by the retained firstinformation is substantially same as the position or the pose of thehand of the target person represented by the retained second informationas the common timing and extracts the pieces of the first and secondinformation acquired at the common timing as the pair information.
 4. Aninformation processing system comprising: a first informationacquisition apparatus configured to acquire first information relatingto at least one of a position or a pose of a hand of a target person; asecond information acquisition apparatus configured to acquire secondinformation relating to at least one of the position or the pose of thehand of the target person and different from the first informationacquired by the first information acquisition apparatus; and aninformation processing apparatus; wherein the information processingapparatus includes an acceptance unit for accepting the firstinformation and the second information from the first and secondinformation acquisition apparatuses, respectively, a retention unit forretaining the accepted first and second information in an associatedrelationship with information of timings at which the first and secondinformation acquisition apparatuses acquire the first and secondinformation, respectively, and an extraction unit for extracting piecesof the first and second information acquired at a common timing from theretained first and second information as pair information; and theextracted pair information is provided for a learning process ofmechanical learning using the first and second information included inthe pair information as a teacher signal and an input signal,respectively, and when it is known that, with regard to a particularposition or pose of the hand of the target person, first informationpieces relating to the particular position or pose and the secondinformation pieces relating to the particular position or pose are notacquired at a common timing, a correction is applied to neverthelessextract the pieces as pair information.
 5. The information processingsystem according to claim 4, wherein the first information acquisitionapparatus detects positions of a plurality of markers arranged on thehand of the target person and acquires the first information relating toat least one of the position or the pose of the hand of the targetperson on a basis of the detected positions of the markers, the secondinformation acquisition apparatus includes a device to be grasped by thehand of the target person and acquires the second information relatingto at least one of the position or the pose of the hand of the targetperson on a basis of information detected by the device, and the markersto be detected by the first information acquisition apparatus arearranged also on the device of the second information acquisitionapparatus, and the first information acquisition apparatus acquiresinformation relating to a position or a posture of the device of thesecond information acquisition apparatus together with the firstinformation relating to at least one of the position or the pose of thehand of the target person and outputs the acquired information to theinformation processing apparatus.
 6. A method in which an informationprocessing apparatus is used that is connected to a first informationacquisition apparatus that acquires first information relating to atleast one of a position or a pose of a hand of a target person and asecond information acquisition apparatus that acquires secondinformation relating to at least one of the position or the pose of thehand of the target person and different from the first informationacquired by the first information acquisition apparatus, the informationprocessing apparatus carrying out a method, comprising: accepting thefirst information and the second information from the first and secondinformation acquisition apparatuses, respectively; retaining theaccepted first and second information in an associated relationship withinformation of timings at which the first and second informationacquisition apparatuses acquire the first and second information,respectively; and extracting pieces of the first and second informationacquired at a common timing from the retained first and secondinformation as pair information, wherein the extracted pair informationis provided for a learning process of mechanical learning using thefirst and second information included in the pair information as ateacher signal and an input signal, respectively, and when it is knownthat, with regard to a particular position or pose of the hand of thetarget person, first information pieces relating to the particularposition or pose and the second information pieces relating to theparticular position or pose are not acquired at a common timing, acorrection is applied to nevertheless extract the pieces as pairinformation.
 7. A non-transitory, computer readable storage mediumcontaining a computer program, which when executed by an informationprocessing apparatus that is connected to a first informationacquisition apparatus that acquires first information relating to atleast one of a position or a pose of a hand of a target person and asecond information acquisition apparatus that acquires secondinformation relating to at least one of the position or the pose of thehand of the target person and different from the first informationacquired by the first information acquisition apparatus, causes theinformation processing apparatus to carry out actions, comprising:accepting the first information and the second information from thefirst and second information acquisition apparatuses, respectively;retaining the accepted first and second information in an associatedrelationship with information of timings at which the first and secondinformation acquisition apparatuses acquire the first and secondinformation, respectively; and extracting pieces of the first and secondinformation acquired at a common timing from the retained first andsecond information as pair information, wherein when it is known that,with regard to a particular position or pose of the hand of the targetperson, first information pieces relating to the particular position orpose and the second information pieces relating to the particularposition or pose are not acquired at a common timing, a correction isapplied to nevertheless extract the pieces as pair information.